VILA: Improving Structured Content Extraction from Scientific PDFs Using Visual Layout Groups

Abstract Accurately extracting structured content from PDFs is a critical first step for NLP over scientific papers. Recent work has improved extraction accuracy by incorporating elementary layout information, example, each token’s 2D position on the page, into language model pretraining. We introduce new methods that explicitly VIsual LAyout (VILA) groups, is, text lines or blocks, to further improve performance. In our I-VILA approach, we show simply inserting special tokens denoting group boundaries inputs can lead 1.9% Macro F1 improvement in token classification. H-VILA hierarchical encoding of layout-groups result up 47% inference time reduction with less than 0.8% loss. Unlike prior layout-aware approaches, do not require expensive additional pretraining, only fine-tuning, which reduce training cost 95%. Experiments are conducted newly curated evaluation suite, S2-VLUE, unifies existing automatically labeled datasets and includes dataset manual annotations covering diverse papers 19 disciplines. Pre-trained weights, benchmark datasets, source code available at https://github.com/allenai/VILA.